A 



Certificate of Express Mailing 

Express Mail Mailing Label No. EK 555 922 541 US 
Date of Deposit: August 20, 2000 

I hereby certify that this paper or fee is being deposited with the United 
States Postal Service "Express Mail Post Office To Addressee" Service 
under 37 CFR 1.10 on the date indicated above and is addressed to the 
Commissioner of Patents and Trademarks, Washington, D. C. 



Attorney Docket No. EMK-001 

Patents 



cvi 

LO : 



^ — 



By: 



Richard A. Jordan 



The Honorable Commissioner of Patents and Trademarks 
U.S. Patent and Trademark Office 
Washington, D.C. 20231 

Dear Sir: 

Please find enclosed a patent application as follows: 

Applicant(s): Shmuel Peleg, Assaf Zomet, Chetan Arora And Takeo Miyazawa 

Title: System And Method For Rectified Mosaicing Of Images Recorded By A Moving 
Camera 

29 Pages Specification, including 27 Claims and Abstract 

14 Sheets Formal Drawings 

Declaration and Power of Attorney 

Assigrmient of invention to: Yissum Research Development Company Of The Hebrew 

University of Jerusalem and Emaki Inc 
Small Entity Declaration 

A check in the amount of $525.00 is attached to cover the fiUng fee. 

X Priority is hereby claimed based on Provisional Appn. Ser. No. 60/149,969, filed August 20, 
1999, and Provisional Appn. Ser, No. 60/168,421, filed November 29, 1999 



Basic Fee 

Additional Fees: 

Total Claims 27 , minus 20 

Total Ind. Claims 6 , minus 3 

Fee for Multiply-Dependent Claims 

Total Filing Fee 



$345.00 

= 7 X $9.00 = $63.00 
= 3 X $39.00 - $117.00 

$0.00 
$525.00 



-1- 



Attorney Docket No. EMK-001 



If this application is found otherwise to be INCOMPLETE, or if at any time it appears that 
a TELEPHONE CONFERENCE with counsel would helpfully advance prosecution, please 
telephone the undersigned in Wellesley, Massachusetts, at (781) 431-1357. 

Kindly acknowledge receipt of the foregoing application by returning the self-addressed 
postcard. 



Respectfully submitted, 

Attorney for Applicant 
Richard A. Jordan 
Reg. No. 27,807 

Richard A. Jordan 
P. O. Box 81363 

Wellesley Hills, MA 02481-0004 



Telephone +1 (781) 431-1357 
Fax+1 (781)235-8326 



ATTORNEYS DOCKET NO. EMK-001 

PATENTS 



United States Patent Application 



Of 



Shmuel Peleg, Assaf Zomet, Cretan Arora And Takeo Miyazawa 



For 



System And Method For Rectified Mosaicing Of Images Recorded By A Moving 



Camera 



Certificate of Express Mailing 

Express Mail Mailing Label No. EK 555 922 541 US 

Date of Deposit August 20, 2000 

I hereby certify that this paper or fee is being deposited 
with the United States Postal Service "Express Mail Post Office 
To Addressee" Service under 37 CFR 1 . 10 on the date indicated 
above and is addressed to the Commissioner of Patents and 
Trademarks, Washington, D. C. 20231. 



Richard A. Jordan 



EMK-001 



1 Field Of The Invention 

2 The invention relates generally to the field of generating mosaic images and more particularly 

3 to generating a rectified mosaic image fi^om a series of images recorded by a moving camera. 

4 Background Of The Invention 

5 In mosaicing of images, a number of overlapping images of a scene are initially recorded by 

6 a camera. Using information in the regions in which the images overlap, a single image is generated 
^ which has, for example, a wider field of view of the scene than might be possible otherwise. 
Si Typically, mosaic images are generated in connection with a plurality of individual images that are 
ft recorded by a camera that is rotated around a stationary optical axis. Such mosaic images provide 
tfb a panoramic view around the optical axis. Additionally, mosaic images are generated firom images 
M recorded by, for example, an aerial camera, translating parallel to the scene and the optical axis is 
12 perpendicular both to the scene and to the direction of camera motion. 

fiB Problems arise, however, in connection with mosaic images that are be made fi-om images 

|b4 recorded by a camera whose optical axis is moved, that is, translated along a particular path and/or 

¥5 rotated around an axis, particularly when different parts of a scene are located at different distances 

1 6 from the camera. When different parts of a scene are located at different distances from the camera, 

1 7 from image to image they appear to move at different rates. That is, when the camera is moved from 

1 8 the location at which one image is recorded, to the location at which the next image is recorded, with 

19 objects in the scene that are close to the camera will move in the image more than objects that are 

20 farther from the camera. Similarly, when the camera is rotated from one angular orientation at which 

2 1 one image is recorded, to another angular orientation at which another image is recorded, obj ects in 

22 the scene whose viewing direction makes a larger angle to the rotation axis will move in the image 

23 more than objects whose viewing direction makes a smaller angle with the rotation axis. In both 
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1 cases, when the images are mosaiced, when corresponding points are located in successive images 

2 and the images are aligned to form the mosaic therebetween, the images will be mosaiced at an 

3 incorrect angle with respect to each other, resulting in a curled mosaic image. 

4 Another problem can arise if, for example, the viewing direction of the camera is not pointed 

5 in a direction generally perpendicular to the direction of motion, but instead at an angle thereto. In 

6 that case, for example, assume that the camera moves to the right and that the camera is pointed 

7 somewhat in the direction of motion, the image contents will generally expand from frame to frame, 

8 as the camera gets closer to the objects in sight. When the second image is warped so that 

9 corresponding points will match the first image, the size of the second image will shrink, resulting 
10 in a mosaic that tapers from left to right. Similarly, when the camera is pointed backward from the 
5 direction of motion, the image contents generally shrink from frame to frame. When the second 
% image is warped so that corresponding points will match the first image, the size of the second image 
% will increase, resulting in a mosaic whose dimensions increase from left to right. 

'^4 Summary Of The Invention 

The invention provides a new and improved system and method for generating a rectified 

56 mosaic image from a series of images recorded by a moving camera. 

17 In brief summary, in one aspect the invention provides a system for generating a rectified 

1 8 mosaic image from a plurality of individual images, the system comprising a quadrangular region 

19 defining module, a warping module and a mosaicing module. The quadrangular region defining 

20 module is configured to define in one individual image a quadrangular region in relation to two 

21 points on a vertical anchor in the one individual image and mappings of two points on a vertical 

22 anchor in at least one other individual image into the one individual image. The warping module 

23 is configured to warp the quadrangular region to a rectangular region. The mosaicing module 

24 configured to mosaic the quadrangular region to the mosaic image. 
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1 In another aspect, the invention provides a system for generating a mosaic from a plurality 

2 of panoramic images, the system comprising a motion determining module, a normahzing module, 

3 a strip selection module, and a mosaicing module. The motion determining module is configured 

4 to determine image motion between two panoramic images. The normalizing module is configured 

5 to normalize respective columns in the panoramic images in relation to the ratio of the image motion 

6 thereof to the image motion of a selected column, thereby to generate normahzed panoramic images. 

7 The strip selection module is configured to select strips of the normalized panoramic images. The 

8 mosaicing module is configured to mosaic the selected strips together. 

9 Brief Description Of The Drawings 

Wb This invention is pointed out with particularity in the appended claims. The above and 

il further advantages of this invention may be better understood by referring to the following 

3^ description taken in conjunction with the accompanying drawings, in which: 

4 3 FIG. 1 schematically depicts a system for generating a rectified mosaic image from a series 

ff4 of images recorded by a moving camera, constructed in accordance with the invention; 

5j5 FIG. 2 schematically depicts operations performed in connection with generating a mosaic 

016 image from a series of individual images; 

17 FIGS. 3 A through 3D are useful in describing a problem that can arise in connection with 

18 generating a mosaic image from a pluraUty of images of a scene using a moving camera; 

1 9 FIGS. 4A through 4D are useful in describing a second problem that can arise in connection 

20 with generating a mosaic image from a plurality of images of a scene using a moving camera; 

2 1 FIG. 5 A through 5D are useful in describing a third problem that can arise in connection with 

22 generating a mosaic image from a plurality of images of a scene using a moving camera; 
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1 FIGS. 6 and 7 are useful in connection with understanding one methodology used by the 

2 system depicted in FIG. 1 in connection with correcting the problem described in connection with 

3 FIGS. 5A through 6D; 

4 FIG. 8 is a flowchart depicting operations performed by the system in connection with the 

5 methodology described in connection with FIGS. 6 and 7; 

6 FIG. 9 is useful in connection with understanding a second methodology used by the system 

7 depicted in FIG. 1 in connection with correcting the problem described in connection with FIGS . 5 A 

8 through 6D; 

9 FIG. 10 is a flowchart depicting operations performed by the system in connection with the 
ffl methodology described in connection with FIGS. 10; 

fl FIGS. 1 1 A through 11 C are useful in connection with operations performed by the system 
in connection with generating a mosaic of panoramic images; and 

X3 FIG. 12 is a flow chart depicting operations performed by the system in connection with 

3;4 generating mosaic panoramic images. 

i Detailed Description of an Illustrative Embodiment 

16 FIG. 1 schematically depicts a system 10 for generating a mosaic image from a series of 

1 7 images recorded by a moving camera, constructed in accordance with the invention. With reference 

1 8 to FIG. 1 , system 1 0 includes a camera 1 1 that is mounted on a rig 12. The camera 1 1 may be any 

1 9 type of camera for recording images on any type of recording medium, including, for example, film, 

20 an electronic medium such as charge-coupled devices (CCD), or any other medium capable of 

21 recording images. The rig 12 facilitates pointing of the camera 11 at a scene 13 to facilitate 

22 recording of images thereof. The rig 1 2 includes a motion control 1 4 configured to move the camera 

23 1 1 . In moving the camera 1 1 , the motion control can translate the camera 1 1 , rotate it around its 
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1 axis, or any combination thereof. In one embodiment, it will be assumed that the motion control 14 

2 can translate the camera 11 along a path 16 and rotate the camera during translation. While the 

3 camera 1 1 is being moved, it can record images 20(1),..., 20(1) (generally identified by reference 

4 numeral 20(i)) of the scene 13 from a series of successive locations along the path 16. The 

5 individual images recorded at the successive locations are provided to an image processor 17 for 

6 processing into a unitary mosaic image, as will be generally described in connection with FIG. 2. 

7 Preferably the successive images 20(i) will overlap, which will facilitate generation of the mosaic 

8 image as described below. 

9 As noted above, the image processor 1 7 processes the series of individual images to generate 
1 0 a unitary mosaic image. Operations performed by the image processor 17 in connection with that 
S operation will generally be described in connection with FIG. 2. With reference to FIG. 2, the image 
% processor 1 7 will initially receive two or more images 20(i). Thereafter the image processor 1 7 will 
% process two individual images, for example, images 20(1) and 20(2) to find overlapping portions 
^4 21(1) and 21(2) of the scene generally toward the right and left sides of the respective images 20(1) 
ifs and 20(2), respectively, and use those corresponding portions 21(1) and 21(2) to aUgn the images 
% 20(1) and 20(2). Given the alignment, portions 21(1) and 21(2) can be defined in images 20(1) and 
fW 20(2) such that they can be combined to form a portion of the mosaic image 22. Thereafter, similar 
33 operations can be performed in connection with the next image 20(3) in the series and the portion 
% of the mosaic image 22 generated using images 20(1) and 20(2) to fiirther extend the mosaic image 

20 22. These operations can be performed in connection with the remaining images 20(4),..., until all 

21 of the images 20(i) have been used to generate the mosaic. 

22 Several problems arise in connection with generation of the mosaic image 21 as described 

23 above. One such problem will be described in connection with FIGS. 3 A through 3D. As noted 

24 above, when the image processor 17 processes two successive individual images 20(1) and 20(2) to 

25 mosaic them together, or an individual image 20(i) (i>2) and the previously-generated mosaic image 

26 22, it uses overlapping portions to aUgn the respective images 20(1) and 20(2). The problem 
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1 described in connection with FIGS. 3A through 3D arises when features that are used to align the 

2 individual images do not have a uniform image motion. This problem can, for example, be 

3 described in connection with a planar surface that is tilted with respect to the camera 1 1 , so that, for 

4 example, the planar surface is parallel to the path 1 6 along which the camera is translated, but tilted 

5 so that the lower portion of the planar surface is closer to the camera 1 1 and the upper portion of the 

6 planar surface farther away from the camera 1 1 . To provide information for aligning the successive 

7 individual images when the mosaic image is generated, the planar surface is provided with a series 

8 of equi-distant vertical lines. 

9 In that case, each of the individual images will appear as the image 30(i) depicted FIG. 3 A, 
10 with the vertical lines appearing as slanted lines 31(1) through 31(S) (generally identified by 
Jl reference numeral 3 l(s)). Since the lower part of the image 30(i) is of the portion of the planar 
% surface that is closer to the camera 1 1 , and the upper part of the image 30(i) is of the portion of the 
% planar surface that is farther from the camera 1 1, the vertical lines on the planar surface will be 
% recorded as the slanted lines 3 1 (s) . The perspective due to the tilting of the planar surface causes the 
'fs Unes that are to the left of the center of the image 30(i) to be slanted toward the right and the Hues 
%> that are to the right of the center to be slanted toward the left in the image 30(i). The distance 
fW between locations along the path 16 at which images 30(i), 30(i+l),... are recorded will be such as 
Ss to allow the portions of the scene recorded in the images to overlap so that, for example, slanted line 
% 31(S-1) in image 30(i+l) corresponds to slanted line 31(2) in image 30(i); otherwise stated, the 

20 image motion, or motion of objects in the successive images from image 30(i) to image 30(i+l), is 

21 such that line 31(S-1) in image 30(i) corresponds to line 31(2) in image 30(i-l). 

22 Conventionally, as described above, when a mosaic image 34 is generated using images 30(i) 

23 as described above, strips 32(i) from the successive images 30(i) will be used as shown in FIG, 3B. 

24 Since line 31(S-1) in image 30(i) corresponds to line 31(2) in the successive image 30(i+l), 

25 conventionally in each image 30(i) the strip 33(i) obtained from the image 30(i) for use in the mosaic 

26 can be defined by lines 31(2) and 31(8-1) in the image 30(i). Accordingly, when the strips 33(i), 
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1 33(i+l) obtained form the two images 30(i) and 30(i+l ) would be aligned to form a mosaic as shown 

2 in FIG. 3B, with the resulting mosaic being curled. The strength of the curl is, in the case of a 

3 rotating camera, a function of the angle between the viewing direction and the rotation axis, and, in 

4 the case of a translating camera, a function of the angle with which the planar surface is tilted with 

5 respect to the image plane of the camera 1 1 . 

6 As will be described below in greater detail, the problem described above in connection with 

7 FIGS. 3A and 3B can be corrected by rectifying the strips 33(i) as shown in FIG. 3C to provide 

8 successive rectangular strips 3 5(i), which, as shown in FIG. 3D, can be mosaiced together to provide 

9 a rectified mosaic image 36. Generally, the orientation of the planar surface relative to the image 
10 plane of the camera 11 is unknown, but the amount of distortion, if any, that is caused by the 
-Si orientation can be determined from the optical flow, that is, the change of the position and 
1=k orientation of respective vertical lines as between successive images. After the amount of distortion 
% has been determined, the image processor 17 can process each of the individual images to correct 

for the distortion prior to integrating them into the mosaic. 

%5 Another problem will be described in connection with FIGS. 4A through 4D. In the situation 

We to be described in connection with FIG. 4, the camera 1 1 is moving from left to right parallel to a 

rni vertical planar surface, and is tilted forward, that is, tilted to the right with respect to the planar 
sxxrface. In that case, the planar surface is provided with Unes running horizontally as well as 

19 vertically, and the camera is tilted with respect to the planar surface in such a manner that it points 

20 to the right. Accordingly, and with reference to FIG. 4A, an image 50(i) recorded by the camera, 

21 the vertical lines 51(1), 51(2),... will remain vertical, but the horizontal lines 52(1), 52(2),... that are 

22 above the horizontal center of the image 50(i) will be angled in a downward direction from left to 

23 right, while horizontal lines 52(H), 52(H-1),... that are below the horizontal center of the image 50(i) 

24 will be angled in an upward direction from left to right. As is the case in connection with image 

25 30(i) (FIG. 3A), line 51(S-1) in image 50(i) corresponds to line 51(2) in image 50(i+l). Thus, in 

26 mosaicing a strip 53(i) from an image 50(i), the strip can be defined by vertical lines 51(2) and 
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1 51(S-1) on the left and right sides, and, for example, by lines 52(1) and 52(H) at the top and the 

2 bottom, in each image 50(i), 50(i+l),.... It will be appreciated that, in matching a strip 53(i+l) from 

3 image 50(i-M) to the left edge of a strip 53(i) from the preceding image 50(i), the strip 53(i+l) will 

4 be warped so that the respective horizontal lines 52(1), 52(2),...,52(H-1), 52(H) along the left edge 

5 of the strip 53(i) will match the same lines 52(1), 52(2),..., 52(H-1), 52(H) along the right edge of 

6 the strip 53(i+l). Accordingly, the mosaic image 54 formed from successive strips will taper from 

7 left to right. As shown in FIG. 4C, the image processor 17 can rectify this distortion by rectifying 

8 each strip 53(i), 53(i+l),... to form rectangular strips 55(i), 55(i+l), with the rectification being such 

9 as to make the lines 52(1), 52(2),..., 52(H-1), 52(H) return to a horizontal orientation and mosaic the 
10 rectangular strips together to form the mosaic image 56 (FIG. 4D). 

5 A third problem, which generally is a combination of those described above in connection 

Hh with FIGS. 3 A through 3D and 4 A through 4D, will be described in connection with FIGS. 5 A 

$3 through 5D. In the problem to be described in connection with FIGS. 5A through 5D, both the 

i|i planar surface comprising the scene and the camera 1 1 recording images of the scene may be tilted. 

% As with the planar surface described above in connection with FIGS. 4A through 4D, the planar 
surface in this case includes a plurality of vertical and horizontal lines. In that case, each image 60(i) 

iV7 as recorded by the camera will as shown in FIG. 5 A, with the region subsumed by the vertical hnes 

Ss 61(1),..., 61 (V) (generally identified by reference numeral 61(v)) tapering vertically from bottom to 

% top (as is the case in the example described above with reference to FIGS . 3 A through 3D), and the 

20 region subsumed by the horizontal lines 62(1),..., 62(H) (generally identified by reference numeral 

21 62(h)) tapering horizontally from left to right. If the motion of camera 1 1 as between successive 

22 images 60(i), 60(i+l),... is such that the vertical line 61(V-1) in image 60(i) corresponds to the same 

23 line in the scene 12 as line 61(2) in image 60(i+l) when the image processor 17 mosaics strips from 

24 the successive images 60(i), 60(i+l),... it can select as the strip the region of each image bordered 

25 by vertical lines 61(1) and 61(2) and horizontal lines 62(1) and 62(H). In that case, if the strip 63(i) 

26 for the mosaic image 64 (reference FIG. 5B) is obtained from image 60(i), since the strip 63(i+l) 

27 from image 60(i+l) will be warped so that the length of its left edge corresponds to the length of the 
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1 right edge of strip 63(i), that strip 63(i+l) will be proportionately smaller and disposed at a different 

2 angle than the strip 63(i) in the mosaic image 64. Each subsequent strip 63(i+2),..., will also be 

3 proportionately smaller than, and disposed at a corresponding angle to, the previous strip 63 (i+1 

4 and so the resulting mosaic image 64 will both be curved and taper from left to right. The image 

5 processor 1 7 can rectify this distortion by rectifying each strip 63(i) both horizontally and vertically 

6 to form a rectangular strip 64(i) (reference FIG. 5C) prior to mosaicing it to the mosaic image 65 

7 (reference FIG. 5D). 

8 Details of how the image processor 17 generally rectifies distortion using information from 

9 apparent motion in successive images will be described in connection with FIGS. 6 through 8. 
10 Generally, as will be appreciated from the above, each strip 70(i-l), 70(i), 70(i+l),... in the 
S mosaic image 71 is obtained from arespective strip 72(i-l), 72(i), 72(i+l),... in successive images 

. 73(i-l), 73(i), 73(i+l),... recorded by the camera 11 . To fully define the transformation to be used 

% for the rectification, the image processor 1 7 will need to define the borders of each strip 70(i) in the 

'iJ4 mosaic image 71 , the borders of the regions in the respective images 73(i) that will comprise for the 

IfS respective strips 72(i) and the mapping transformation from the strip 72(i) to the strip 70(i). To 

ft accompUsh that, the image motion between successive pairs of images 73(i-l), 73(i) and 73(i), 

ni7 73(i+l) is determined. Generally, for image 73(i), the image processor 17 defines the region that 

38 is to comprise the strip 72(i) to satisfy three conditions, namely: 

1 9 (i) One border 74(i)(l) of the region should match the border 74(i-l) of the region of image 

20 73 (i- 1 ) that is to comprise strip 72(i- 1 ) in the preceding image 73 (i- 1 ), which will map to the border 

21 75(i-l) between strips 70(i-l) and 70(i) in the mosaic image 71; 

22 (ii) The border 74(i)(2) of the region that is to comprise the strip 72(i), which will map to the 

23 border 75(i) between strips 70(i) and 70(i+l) in the mosaic image 7 1 , is chosen such that the distance 

24 between the two borders 74(i)(l) and 74(i)(2) is proportional to the image motion at each border 

25 location; this will ensure that the mosaic image 71 is constructed hnearly and not cinved; and 
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1 (iii) The top and bottom borders 76(i) and 77(i) of the region of image 73 (i) that is to 

2 comprise strip 72(i) should pass through the top and bottom ends of some vertical column in the 

3 image 73(i), such as the vertical column at the center of the image 73(i); this will ensure that the strip 

4 70(i) is not expanded or shrunk in the mosaic image 7L 

5 The rectangular strips 70(i-l), 70(i), 70(i+l),.„ in the mosaic 71 have a uniform height 

6 to provide a mosaiced image 71 of uniform height and to avoid expanding or shrinking the mosaic 

7 image 71 vertically. However, the vertical location of the strip 70(i) in the mosaic image 7 1 changes 

8 according to the vertical motion or tilt of the camera 1 1 . The width of the strip 70(i) is determined 

9 by the motion of the scene 12 from image to image. 

ap In the case images recorded of a scene comprising a planar surface, or images recorded by 

M a tilted rotating camera, operations performed by the image processor 17 in one embodiment in 

^ generating the strip 70(i) to be used in generating the mosaic image 71 will be described in 

ij3 connection with FIG. 7 and the flowchart in FIG. 8. Generally, in the embodiment described in 

^ connection with FIGS. 7 and 8, the image processor 17 obtains the strip as defined one side of an 

!L5 vertical anchor. The vertical anchor is a vertical feature in the image that remains invariant under 

Q06 the transformation that warps a strip in the image to a strip in the mosaic. Only transformations that 

r|7 keep the anchor invariant will be considered for warping a strip in the image to a strip in the mosaic. 

^8 The vertical anchor may be anywhere in the image 73(i), illustratively, the center, the left border, 

1 9 or other column; in the embodiment described in connection with FIGS . 7 and 8, the vertical anchor 

20 is selected to be the left border of the image 73(i), and that vertical anchor will also form the left 

21 border of the strip 70(i). Another embodiment, in which the strip is defined on two sides of a 

22 vertical anchor, will be described below in connection with FIGS. 10 and 1 1 . 

23 With reference to FIGS. 7 and 8, the image processor 17 will initially locate the vertical 

24 anchor in the image 73(i) (step 1 00) and identify the points Pk and (where index "k" has the same 

25 value as index "i") at which the anchor intersects with the top and bottom borders of the image 73(i) 

26 (step 101). Using the homography U^'^ between images 73 (i+ 1 ) and 73 (i), the image processor also 
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maps the points Pk+i and in image 73(i+l) to image 73(i) as points ?\ and Q\, respectively (step 
102). It will be appreciated that points Pj^+i and Q^+i comprise, respectively, the points at which the 
anchor in image 73(i+l), that is, the left border of the image 73(i), intersects the top and bottom of 
that image 73(i+l), and so F\= and Q~kTik ^(Qk+i)- 

After the image processor 17 has located points ?\ and Q\, it identifies the line passing 
therethrough (step 103) and then identifies two points and QV on the line such that the distance 
between them along the line is the same as the distance between points P~k and Q\, and their 
centroid is in the middle row of the image (step 1 04). The image processor can determine the region 
of image 73(i) that is to be used as the strip 72(i) is the quadrilateral defined by points P'k, QV, Qk 
and Pk (step 105) and warp the strip to rectangular form using a smooth (for example, bilinear) 
interpolation of the coordinates of those points, thereby to generate the strip 70(i) (step 106). It will 
be appreciated that the use of an interpolation is an approximation of the real transformation, which 
is unknown, but if the strip 72(i) is relatively narrow, the approximation will suffice. Thereafter, the 
image processor 17 can mosaic the strip 70(i) to the previously-generated mosaic image 71, if any 
(step 107). 

In addition, the image processor 17 will determine the vertical offset to be used for the next 
strip 70(i+l) (step 108). In that operation, the image processor will determine the vertical offset as 

h 

Qk ~ Q\ * ' where ||A-B|| refers to the distance between two points A and B and "h" 



is the image height. 

As noted above, a second embodiment, in which the strip is defined on two sides of a vertical 
anchor, is described in connection with FIGS. 9 and 10. The vertical anchor may be any column 
in the image 73(i); in one embodiment is selected to be the center colimm, since that will reduce lens 
distortion. In this embodiment, the image processor identifies two regions, approximately symmetric 
on opposing sides of the center column, both of which be warped to form respective portions of the 
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1 strip 70(i) to be used in the mosaic image 7 1 . With reference to FIGS . 9 and 1 0, the image processor 

2 17 will initially identify the vertical anchor in the image 73(i) (step 120) and identify the points 

3 and at which the anchor intersects with the top and bottom borders of the image 73 (i) (step 121). 

4 Thereafter, the image processor 17 will determine a value for "d," the vertical offset between the 

5 point Ok_i that comprises the center of the image 73(i-l), that is, the projection of point O^.y in image 

6 73, that is, Ht.i(Ok.i) (step 122), where Hk.i is the homography between image 73(i-l) and image 

7 73(i), and identify two points PV and Q'j, which correspond to points P^ and shifted vertically by 

8 an amount corresponding to the value "d" (step 123). The image processor 17 will perform 

9 operations similar to steps 122 and 123 as between images 73(i) and 73(i+l) using the homography 
1 0 Hk therebetween (step 1 24), 

J In addition, the image processor 17, using the homography Hk.i, maps the points P^.i andQ^.i 

^ to image 73(i) as points Hk.i(Pk-i) and Hk.i(Qk.i), respectively (step 125), and, using the homography 

fte Hk^i, maps points P^+i and Q'k+i to image 73 (i) as points H"\(P'k+i) and H"^k(Qk+i) (step 126). The 

% points Hk.i(Pk.i), P'k. Q and Hk.i(Qk.i) define a left quadrangular region 80(i)(L), and points P^, 

T5 H"\(Pk+i), H'\(Qk.i) and Qk define a right quadrangular region 80(i)(R), a portion of each of which 

g6 will be used in generating respective rectangular portions 8 1 (i)(L) and 8 1 (i)(R) that together will be 

r||7 used as the strip for the image 73(i) in the mosaic image 71 . Essentially, it will be desired to use left 

SS quadrangular region 80(i)(L), along with the right quadrangular region 80(i-l)(R) associated with 

^b the previous image 74(i-l), in connection with a rectangular region 81(j) in the mosaic image 71. 

20 Similarly, it will be desired to use the right quadrangular region 80(i)(R), along with the left 

21 quadrangular region 80(i+l)(L) associated with the next image 73(i+l), in connection with the next 

22 rectangular region 81(j+l) in the mosaic image 71 . The size and shape of the respective rectangular 

23 regions is somewhat arbitrary. Since both images 73(i-l) and 73 (i) are used to provide half of the 

24 image to be used in the rectangular region 8 1 (j), it will be appreciated that the points Hk.i(Pk_i), PV, 

25 Q k, and Hk.i(Qk.i) that define the left quadrangular region 80(i)(L) will also relate to the points 

26 defining the comers of the rectangular region 8 1 (j), and it will be necessary to find the points i and 

27 A21 that relate to the mid-points of the top and bottom of the rectangular region 8 l(j), respectively. 
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Accordingly, the portion of quadrangular region 80(i)(L) that will be used in connection with the 
left-hand portion of strip 70(i) is the quadrangular region 82(i) defined by points An, P^, QV, and 
A21. Similarly, points Pk, H-^k(Pk+i), H-\(Qk.i) and Qi, that define the right quadrangular region 
80(i)(R) will also relate to the points defining the comers of the rectangular region 81(j+l), and it 
will be necessary to find the points A^ and A22 that relate to the mid-points of the top and bottom 
of the rectangular region 81(j+l), respectively. Accordingly, the portion of quadrangular region 
80(i)(R) that will be used in connection with the left-hand portion of strip 70(i) is the quadrangular 
region 83(i) defined by points P^, A12, A22 and Qk. The rectangular regions 81(j) and 81(j+l) can 
both be defined by points UVWX, with points U and V defining the left and right top comers, 
respectively, and points W and X defining the right and left bottom comers, respectively. In that 
case the relationship between the left and right quadrangular regions 80(i)(L) and 80(i)(R) will be 
defined by respective homographies Fl and Fr. 



Accordingly, following step 1 26, the image processor 1 7 will identify the points A^ , A21 , A12 



and A22 as 



4i = 



(U+V 



U+V 



Ax = Fl 



Ai = Fr\ 



K 2 J 



(1) 



(step 127), and warp the portion of the quadrangular region defined by points An, P'k, QV 2nd A21 
to the right portion of the rectangular region 8 1 (j) and the portion of the quadrangular region defined 
by points A12, Pk, Qk and A22 to the left portion of the rectangular region 81(j+l)bya smooth (for 
example, bilinear) interpolation thereby to provide respective rectangular portions 70(i)(L) and 
70(i)(R) of the strip 70(i) associated with image 73 (i), with the rectangular portion 81(i)(R) being 
vertically offset from rectangular portion 8 1 (i)(L) by the value "d" determined in step 1 22 (step 128). 
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1 In the system 10 as described above in connection with FIGS. 1 through 10, the camera 1 1 

2 has been one that records images in a particular direction. As a further aspect of the invention, the 

3 image processor 17 can also generate a mosaic of a plurahty of panoramic images. Typically, a 

4 panoramic image is generated from a plurality of images recorded from a number of angular 

5 orientations around a common axis, which images are mosaiced together to provide a single 

6 panoramic image of the scene surrounding the axis. The panoramic image so generated is typically 

7 the full 360 degree circle surrounding the axis, or a substantial part thereof The images that are used 

8 in generating the panoramic image may be recorded by a single camera that is rotated around the axis 

9 to facilitate recording of the images from the requisite plurahty of angular orientations, or by a 
1 0 plurality of cameras disposed at the requisite angular orientations. A panoramic image can also be 
© obtained by a single camera with a very wide field of view, which may be provided by a very wide 

angle lens, a combination of lenses and mirrors, or other arrangements as will be apparent to those 

3p skilled in the art. The panoramic image may be cylindrical or alternatively it may be flat. In 

yft accordance with this aspect of the invention, the images are recorded to facihtate generation of a 

f|j^ plurality of panoramic images recorded at successive locations along the axis, with the panoramic 

%6 images overlapping such that the image processor 17 can mosaic them together along the direction 

WJ of the axis. 

Js This aspect will be described in greater detail in connection with FIGS. 1 1 A through 12. 

4^ With reference to FIG. 1 1 A, that FIG. schematically depicts a train tuimel 90 having left and right 

20 sides 91 and 92, a floor 93 and a ceihng 94. The left and right sides 91 and 92 and the floor are 

2 1 planar surfaces, and the ceiling 94 is arched. A pair of tracks 95 is disposed on the floor to facilitate 

22 traversal of the tunnel by a train (not shown). A panoramic camera 96, comprising, for example, a 

23 plurahty of individual cameras disposed around a common axis 97, which extends generally parallel 

24 to the length of the tunnel, records images along the axis from a plurality of angular orientations. 

25 The camera 96 is moved along the axis 97 to facilitate recording of images from which a series of 

26 panoramic images along the axis 97 can be generated, which series can be processed as described 
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1 below in connection with FIG. 13, and the processed panoramic images 100(1), 100(2),... mosaiced 

2 together to form a single mosaic panoramic image 100 (FIG. 1 IC). 

3 As noted above, the tunnel 90 comprises left and right sides 91 and 92, a floor 93 and a 

4 ceiling 94. In the following, it will be assumed that the siirface of the ceiling 94 is cylindrical with 

5 an axis corresponding to the axis 97. In addition, it will be assumed that the distance from axis 97 

6 to each of the left and right sides 91 and 92 and floor 93 is smallest at the center of the left and right 

7 sides and floor, and largest at the comers. In that case, the image motion, that is, the apparent 

8 motion of features and obj ects in the images as between panoramic images will be as depicted in the 

9 graph depicted FIG. 1 IB. With reference to FIG. 1 IB, since, for each of the left and right sides 9 1 
10 and 92 and floor 93, the distance from the axis 97 thereto increases from the center to the two 
!Sl comers, the image motion decreases from the center to the two comers, as shown in left, bottom and 

right graph segments 1 0 1 , 1 02 and 1 03 in FIG. 1 1 B . On the other hand, since, for the ceiUng 94, the 

flB distance from the axis is constant, the image motion will also be constant, as shown in the top graph 

4|4 segment 104 in FIG. IIB. If the internal parameters of the camera 96 are known it will be 

'fS appreciated that the shape of the tuimel 90, to a scale factor, can readily be determined using the 

36 image motion. In addition, given certain other information, such as the distance from the axis 97 

f§7 to the tracks 95, the scale factor can also be determined. 

tfS The image processor 17, in generating a mosaic panoramic image 101 from the individual 

19 panoramic images 100(1), 100(2),..., will process the individual panoramic images to correct for the 

20 differences in the image motion. Operations performed by the image processor 17 in generating a 

2 1 mosaic panorama image 1 00 will be described in connection with the flow chart in FIG. 1 1 . With 

22 reference to FIG. 12, after the image processor 17 has generated or otherwise obtained two 

23 successive panoramic images lOO(i), 100(i+l) that are to be mosaiced together (step 150), for each 

24 column it determines the image motion between the two panoramic images (step 151). The image 

25 motion as determined by the image processor 1 7 may have a motion profile similar to that described 
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1 above in connection with FIG. 13, with image motion of regions relatively close to the camera being 

2 relatively high and image motion of regions further away being relatively low. 

3 Thereafter, the image processor 1 7 normahzes respective colxmms in each panoramic image 

4 1 00(i), 1 00(i+ 1 ) by stretching them in relation to the ratio of the image motion associated with that 

5 column to the image motion of a pre-selected column (step 152), each column comprising the series 

6 of picture elements in the direction parallel to the axis 97. The pre-selected column may be the 

7 column with the highest motion, or any other selected column in the panoramic image. Preferably, 

8 in performing step 152, the image processor 17 will leave at least one row or set of colunms 

9 unchanged. If, for example, the image processor 1 7 does not normaUze the colimms of the portion 
10 of the panoramic image relating to the floor 91, in the resulting mosaic panoramic image the floor 
M will appear to be flat and the ceiling 94 will appear to be curved. On the other hand, if the image 
Hh. processor 1 7 does not normaUze the columns of the portion of the panoramic image relating to the 

ceiling 94, in the resulting mosaic panoramic image the ceiling will appear to be flat and the floor 

%$■ will appear to be curved. Similarly, if the image processor does not normalize the columns of the 

T5 portion of the panoramic image relating to the left and/or right sides, in the mosaic panoramic image 

the left and/or right sides will appear to be flat and both the ceiUng and floor will appear to be 

flJ7 curved. 

^ After the image processor 17 has normalized the respective panoramic images lOO(i), 

19 100(i+l) (step 152), it will select parallel strips therein (step 153) and mosaic the parallel strips into 

20 the mosaic image 100 (step 154). 

21 The system 10 provides a number of advantages. In one aspect, the system provides an 

22 arrangement that can generate mosaic images of scenes including tilted surfaces using a translated 

23 camera that is pointed toward the scene generally sideways. In this aspect, the camera may be 

24 translated in a direction that is parallel to the tilted surface and pointed directly thereat, that is, 

25 perpendicular to the translation direction (reference FIGS. 3A through 3D). Alternatively, and more 
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1 generally, the camera may be pointed in a direction that is tihed with respect to the direction of 

2 motion (reference FIGS. 7A through 10). 

3 In another aspect, the system 1 0 can generate a mosaic of panoramic images (reference FIGS . 

4 11 A through 12) and in connection therewith can determine the shapes of surfaces in the mosaic 

5 images. In connection with this aspect, although the system 1 0 was described as generating a mosaic 

6 of panoramic images of a train tunnel, it will be appreciated that the system can generate such a 

7 mosaic panoramic image of a variety of kinds of scenes, including but not limited to water or sewer 

8 pipes, corridors and hallways, and the like. 

9 It will be appreciated that a number of modifications may be made to the system 10 as 
fi) described above. For example, it will be appreciated that, if the camera 1 1 is translated, it may be 

translated in any direction with respect to the scene 1 3 . In addition, although the system 1 0 has been 

ft described as performing operations in connection with a scene 1 3 that has vertical and/or horizontal 

UB lines, it will be appreciated that the operations can be performed in connection with any pattern or 

^ set of points that ^pear along such lines. 

It will be appreciated that a system in accordance with the invention can be constructed in 
whole or in part from special purpose hardware or a general purpose computer system, or any 
tf7 combination thereof; any portion of which may be controlled by a suitable program. Any program 
98 may in whole or in part comprise part of or be stored on the system in a conventional manner, or it 

19 may in whole or in part be provided in to the system over a network or other mechanism for 

20 transferring information in a conventional manner. In addition, it will be appreciated that the system 

2 1 may be operated and/or otherwise controlled by means of information provided by an operator using 

22 operator input elements (not shown) which may be connected directly to the system or which may 

23 transfer the information to the system over a network or other mechanism for transferring 

24 information in a conventional manner. 



-17- 



EMK-001 

The foregoing description has been hmited to a specific embodiment of this invention. It will 
be apparent, however, that various variations and modifications may be made to the invention, with 
the attainment of some or all of the advantages of the invention. It is the object of the appended 
claims to cover these and such other variations and modifications as come within the true spirit and 
scope of the invention. 

What is claimed as new and desired to be secured by Letters Patent of the United States is: 
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Claims 

1 LA system for generating a mosaic image from a plurality of individual images comprising: 

2 A. a quadrangular region defining module configured to define in one individual image a 

3 quadrangular region in relation to two points on a vertical anchor in the one individual image 

4 and mappings of two points on a vertical anchor in at least one other individual image into 

5 said one individual image; 

6 B. a warping module configured to warp the quadrangular region to a rectangular region; and 
, J C. a mosaicing module configured to mosaic the quadrangular region to the mosaic image. 

2. A system as defined in claim 1 in which said two points on said vertical anchor in said one 
''^2 individual image comprise points at which the vertical anchor intersects upper and lower borders of 
3 said one individual image. 

□1 3 . A system as defined in claim 1 in which the warping module is configured to smoothly interpolate 
""2 between said quadrangular region and said rectangular region. 

1 4. A system as defined in claim 1 in which said quadrangular region defining module includes: 

2 A. a first vertical anchor point identifying module configured to identify two points on the 

3 vertical anchor in said one individual image; 
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4 B. a second vertical anchor point identifying module configured to identify two points in said 

5 one individual image at which the two points on the vertical anchor in said at least one other 

6 individual image map to said one individual image; 

7 C . a first hne segment determining module configured to determine a hne segment between the 

8 two points in said one individual image at which the two points on the vertical anchor in said 

9 at least one other individual image map to said one individual image; and 

10 D. a second line segment determining module configured to determine a second hne segment 

1 1 along a hne defined by the first line segment, the second line segment being such that the 

12 distance between the end points corresponds to the distance between the end points of the 
m first line segment and positioned along the line such that the centroid of the end points of the 

second line segment corresponds to the vertical center of the one image, 

te the quadrangular region corresponding to the two points on the vertical anchor in said one individual 

Cl6 image and the end points of said second line segment. 

m 5. A system as defmed in claim 4 further including a vertical offset value generating module 
configured to generate an offset value for use in the mosaicing step for a rectangular region 

^^3 generated for said other individual image, 

1 6. A system as defmed in claim 1 in which the quadrangular region defining module is configured 

2 to define two quadrangular regions on opposmg sides of the vertical anchor, each in relation to 

3 mappings of two points on a vertical anchor in at least two other individual images into said one 

4 individual image. 
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1 7. A system as defined in claim 6 in which the quadrangular region defining module is configured 

2 to define a left quadrangular region in relation to 

3 (i) mappings of the two points on the vertical anchor in a left individual image into said one 

4 individual image, and 

5 (ii) two points on the vertical anchor in said one individual image, shifted vertically in relation 

6 to a vertical offset between the center of the one individual image and the mapping of the 

7 center of the left individual image to the one individual image, 

a 8. A system as defined in claim 6 in which the quadrangular region defining module is configured 

^ to define a right quadrangular region in relation to 

fLS (i) mappings of the two points on the vertical anchor in a left individual image into said one 
Sft- individual image and 

1,5 (ii) said two points on the vertical anchor in said one individual image. 

'^1 9. A system of generating a mosaic firom a plurality of panoramic images comprising: 

2 A. a motion determining module configured to determine image motion between two panoramic 

3 images; 

4 B . a normahzing module configured to normalize respective columns in said panoramic images 

5 in relation to the ratio of the image motion thereof to the image motion of a selected column, 

6 thereby to generate normalized panoramic images; 

7 C . a strip selection module configured to select strips of the normalized panoramic images; and 
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8 D. a mosaicing module configured to mosaic the selected strips together. 

1 10. A method of generating a mosaic image from a plurality of individual images comprising the 

2 steps of: 

3 A. a quadrangular region defining step in which, in one individual image a quadrangular region 

4 is defined in relation to two points on a vertical anchor in said one individual image and 

5 mappings of two points on a vertical anchor in at least one other individual image into said 

6 one individual image; 

oZ B . a warping step in which the quadrangular region is warped to a rectangular region; and 
C. a mosaicing step in which the quadrangular region is mosaiced to the mosaic image. 

1"1 1 1. A method as defined in claim 10 in which said two points on said vertical anchor in said one 
5^ individual image comprise points at which the vertical anchor intersects upper and lower borders of 
fli3 said one individual image. 

1 12. A method as defined in claim 10 in which the warping step includes the step of smoothly 

2 interpolating between said quadrangular region and said rectangular region. 

1 13. A method as defined in claim 10 in which said quadrangular region defining step includes the 

2 steps of: 

3 A. identifying the two points on the vertical anchor in said one individual image; 
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4 B. identifying two points in said one individual image at which the two points on the vertical 

5 anchor in said at least one other individual image map to said one individual image; 

6 C . determining a Une segment between the two points in said one individual image at which the 

7 two points on the vertical anchor in said at least one other individual image map to said one 

8 individual image; and 

9 D. determining a second line segment along a line defined by the first line segment, the second 

1 0 line segment being such that the distance between the end points corresponds to the distance 

1 1 between the end points of the first line segment and positioned along the line such that the 

1 2 centroid of the end points of the second line segment corresponds to the vertical center of the 
p one image, 

the quadrangular region corresponding to the two points on the vertical anchor in said one individual 

% image and the end points of said second line segment. 

gl 1 4. A method as defined in claim 1 3 further including a vertical offset value generating step in which 

n2 an offset value is generated for use in the mosaicing step for a rectangular region generated for said 

p3 other individual image. 

1 1 5 . A method as defined in claim 1 0 in which the quadrangular region defining step includes the step 

2 of defining two quadrangular regions on opposing sides of the vertical anchor, each in relation to 

3 mappings of two points on a vertical anchor in at least two other individual images into said one 

4 individual image. 
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1 1 6. A method as defined in claim 1 5 in which the step of defining two quadrangular regions includes 

2 a left quadrangular defining step in which a left quadrangular region is defined in relation to 

3 (i) mappings of the two points on the vertical anchor in a left individual image into said one 

4 individual image, and 

5 (ii) two points on the vertical anchor in said one individual image, shifted vertically in relation 

6 to a vertical offset between the center of the one individual image and the mapping of the 

7 center of the left individual image to the one individual image. 

a 1 7. A method as defined in claim 1 6 in which the step of defining two quadrangular regions includes 

52 a right quadrangular defining step in which a right quadrangular region is defined in relation to 

f^J (i) mappings of the two points on the vertical anchor in a left individual image into said one 
S4 individual image and 

L5 (ii) said two points on the vertical anchor in said one individual image. 

1 1 8 . A method of generating a mosaic from a plurality of panoramic images comprising the steps of: 

2 A, a motion determining step in which image motion is determined between two panoramic 

3 images; 

4 B . a normalizing step in which respective columns in said panoramic images are normahzed in 

5 relation to the ratio of the image motion thereof to the image motion of a selected column, 

6 thereby to generate normahzed panoramic images; 

7 C. a strip selection step in which strips of the normahzed panoramic images are selected; and 
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8 D. a mosaicing step in which the selected strips are mosaiced together, 

1 19. A computer program product for use with a computer to provide a system for generating a 

2 mosaic image from a plurality of individual images, the computer program product comprising a 

3 machine readable medium having encoded thereon: 

4 A, a quadrangular region defining module configured to enable the computer to define in one 

5 individual image a quadrangular region in relation to two points on a vertical anchor in said 

6 one individual image and mappings of two points on a vertical anchor in at least one other 

7 individual image into said one individual image; 

^ B. a warping module configured to enable the computer to warp the quadrangular region to a 

!f9 rectangular region; and 

W C. a mosaicing module configured to enable the computer to mosaic the quadrangular region 

Tl to the mosaic image. 

gl 20. A system as defined in claim 19 in which said two points on said vertical anchor in said one 

^ individual image comprise points at which the vertical anchor intersects upper and lower borders of 

3 said one individual image. 

1 21. A system as defined in claim 19 in which the warping module is configured to enable the 

2 computer to smoothly interpolate between said quadrangular region and said rectangular region. 

1 22. A system as defined in claim 19 in which said quadrangular region defining module includes: 
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2 A. a first vertical anchor point identifying module configured to enable the computer to identify 

3 two points on the vertical anchor in said one individual image; 

4 B. a second vertical anchor point identifying module configvired to enable the computer to 

5 identify two points in said one individual image at which the two points on the vertical 

6 anchor in said at least one other individual image map to said one individual image; 

7 C. a first line segment determining module configured to enable the computer to determine a 

8 line segment between the two points in said one individual image at which the two points on 

9 the vertical anchor in said at least one other individual image map to said one individual 
10 image; and 

Jl D. a second line segment determining module configured to enable the computer to determine 

^ a second line segment along a line defined by the first line segment, the second line segment 

% being such that the distance between the end points corresponds to the distance between the 

%4 end points of the first line segment and positioned along the line such that the centroid of the 

J5 end points of the second line segment corresponds to the vertical center of the one image, 

the quadrangular region corresponding to the two points on the vertical anchor in said one individual 

[§7 image and the end points of said second line segment. 

1 23. A system as defined in claim 22 further including a vertical offset value generating module 

2 configured to enable the computer to generate an offset value for use in the mosaicing step for a 

3 rectangular region generated for said other individual image. 

1 24. A system as defined in claim 1 9 in which the quadrangular region defining module is configured 

2 to enable the computer to define two quadrangular regions on opposing sides of the vertical anchor, 
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3 each in relation to mappings of two points on a vertical anchor in at least two other individual images 

4 into said one individual image. 

1 25 . A system as defined in claim 24 in which the quadrangular region defining module is configured 

2 to enable the computer to define a left quadrangular region in relation to 

3 (i) mappings of the two points on the vertical anchor in a left individual image into said one 

4 individual image, and 

5 (ii) two points on the vertical anchor in said one individual image, shifted vertically in relation 
a to a vertical offset between the center of the one individual image and the mapping of the 

center of the left individual image to the one individual image. 

nl 26. A system as defined in claim 25 in which the quadrangular region defining module is configured 

L2 to enable the computer to define a right quadrangular region in relation to 

^13 (i) mappings of the two points on the vertical anchor in a left individual image into said one 
p4 individual image and 

5 (ii) said two points on the vertical anchor in said one individual image. 

1 27. A system of generating a mosaic from a pluraUty of panoramic images comprising: 

2 A. a motion determining module configured to enable the computer to determine image motion 

3 between two panoramic images; 
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4 B. a normalizing module configured to enable the computer to normalize respective columns 

5 in said panoramic images in relation to the ratio of the image motion thereof to the image 

6 motion of a selected column, thereby to generate normalized panoramic images; 

7 C. a strip selection module configured to enable the computer to select strips of the normalized 

8 panoramic images; and 

9 D. a mosaicing module configured to enable the computer to mosaic the selected strips together. 
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1 Abstract Of The Disclosure 

2 A system is described for generating a rectified mosaic image from a plurality of individual 

3 images, the system comprising a quadrangular region defining module, a warping module and a 

4 mosaicing module. The quadrangular region defining module is configured to define in one 

5 individual image a quadrangular region in relation to tvv^o points on a vertical anchor in the one 

6 individual image and mappings of two points on a vertical anchor in at least one other individual 

7 image into the one individual image. The warping module is configured to warp the quadrangular 

8 region to arectangular region. The mosaicing module configured to mosaic the quadrangular region 

9 to the mosaic image. The system fiirther generates a mosaic from a plurality of panoramic images, 
©) the system comprising a motion determining module, a normahzing module, a strip selection 
m module, and a mosaicing module. The motion determining module is configured to determine image 
J2 motion between two panoramic images. The normalizing module is configured to normalize 
^ respective columns in the panoramic images in relation to the ratio of the image motion thereof to 
nH the image motion of a selected column, thereby to generate normalized panoramic images. The strip 

selection module is configured to select strips of the normalized panoramic images. The mosaicing 

^ module is configured to mosaic the selected strips together. 
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100. IMAGE PROCESSOR 17 LOCATES A VERTICAL 
ANCHOR IN THE IMAGE 73(1) 



] 



j 



101 . IMAGE PROCESSOR 17 IDENTIFIES THE POINTS AT 
WHICH THE ANCHOR INTERSECTS WITH THE TOP AND 
BOTTOM BORDERS OF THE IMAGE 73(1) 



T 



102. IMAGE PROCESSOR 17 MAPS THE POINTS AT WHICH 
ANCHOR IN IMAGE 73(1+1) INTERSECTS TOP AND 
BOTTOM BORDERS OF IMAGE 73(i+1) TO IMAGE 73(i) 



103. IMAGE PROCESSOR 17 IDENTIFIES THE LINE 
PASSING THROUGH POINTS MAPPED IN STEP 102 



I 



/T04. IMAGE PROCESSOR 17 IDENTIFIES TWO POINTS ON 
THE LINE IDENTIFIED IN STEP 103 SUCH THAT THE 
DISTANCE BETWEEN THEM ALONG THE LINE IS THE 
SAME AS THE DISTANCE BETWEEN POINTS MAPPED IN 

ySTEPIO^^^^^^^^ 



105. IMAGE PROCESSOR 17 IDENTIFIES THE REGION OF 
IMAGE 73(1) THAT IS TO BE USED AS THE STRIP 72(i) IS 
THE QUADRILATERAL DEFINED BY POINTS POINTS 
IDENTIFIED IN STEPS 101 AND 104 



FTG, 8 
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106. IMAGE PROCESSOR 17 WARPS THE REGION 
IDENTIFIED IN STEP 105 TO RECTANGULAR FORM USING 
A SMOOTH 



I 



107. IMAGE PROCESSOR 17 MOSAICS THE WARPED 
REGION GENERATED IN STEP 106 TO THE 
PREVIOUSLY-GENERATED MOSAIC IMAGE 71, IF ANY 



108. IMAGE PROCESSOR 17 DETERMINES THE VERTICAL 
OFFSET TO BE USED FOR THE NEXT STRIP 
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120. IMAGE PROCESSOR 17 INITIALLY IDENTIFIES THE 
VERTICAL ANCHOR IN THE IMAGE 73(i) 



I 



121. IMAGE PROCESSOR 17 IDENTIFIES THE POINTS AT 
WHICH THE ANCHOR INTERSECTS WITH THE TOP AND 
BOTTOM BORDERS OF THE IMAGE 73(i) 



122. IMAGE PROCESSOR 17 DETERMINES A VALUE FOR 
"D," THE VERTICAL OFFSET BETWEEN THE CENTER OF 
THE IMAGE 73(i-1) AND THE CENTER OF THE IMAGE 73(i) 



123. IMAGE PROCESSOR 17 IDENTIFIES TWO POINTS 
WHICH CORRESPOND TO POINTS IDENTIFIED IN STEP 
121, SHIFTED VERTICALLY BY AN AMOUNT 
CORRESPONDING TO THE VALUE "d" 



124. IMAGE PROCESSOR 17 PERFORMS OPERATIONS 
SIMILAR TO STEPS 122 AND 123 AS BETWEEN IMAGES 
73(1) AND 73(1+1) 



J 



^25. IMAGE PROCESSOR 17 MAPS THE POINTS AT WHICH^ 
THE VERTICAL ANCHOR IN IMAGE 73(i-1) INTERSECTS 
THE BOTTOM AND BORDERS OF IMAGE 73(1-1) TO IMAGE 
73(1) 



K F/(7. 10 
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^^26. IMAGE PROCESSOR 17 MAPS THE POINTS AT WHICH^ 
THE VERTICAL ANCHOR IN IMAGE 73(1-1) INTERSECTS 
THE BOTTOM AND BORDERS OF IMAGE 73(1+1) TO IMAGE | 

^ 



127. IMAGE PROCESSOR 17 IDENTIFIES THE POINTS TO 
DEFINE RESPECTIVE LEFT AND RIGHT QUADRANGULAR 
REGIONS 82(0 AND 83(1) 



] 



128. IMAGE PROCESSOR 17 WARPS THE 
QUADRANGULAR REGIONS TO RECTANGULAR FORM 
AND MOSAICS THEM TO THE PREVIOUSLY-GENERATED 
MOSAIC IMAGE 71, IF ANY 



FTG. lOA 



13/14 
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150. IMAGE PROCESSOR 17 GENERATES OR OTHERWISE 
OBTAINED TWO SUCCESSIVE PANORAMIC IMAGES 



] 



151. IMAGE PROCESSOR 17, FOR EACH COLUMN, 
DETERMINES THE IMAGE MOTION BETWEEN THE TWO 
PANORAMIC IMAGES 



t^52. IMAGE PROCESSOR 1 7 NORMALIZES RESPECTIVE 
COLUMNS IN EACH PANORAMIC IMAGE BY STRETCHING 
THEM IN RELATION TO THE RATIO OF THE IMAGE 
MOTION ASSOCIATED WITH THAT COLUMN TO THE 
IMAGE MOTION OF A PRE-SELECTED COLUMN 



153. IMAGE PROCESSOR 17 SELECTS PARALLEL STRIPS 
THEREIN 



) 



154. IMAGE PROCESSOR 17 MOSAICS THE SELECTED 
PARALLEL STRIPS INTO THE MOSAIC IMAGE, IF ANY. 



